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Some  Logistics  and  Guidelines 


Restroom  location? 


Beverages  and  snacks? 


Lunchtime  arrangements? 
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Some  Logistics  and  Guidelines 


Let’s  return  from  breaks  on  time. 


Let’s  stay  focused.  Please  shut  your  phone  off 
or  enable  silent  mode.  Please  handle  email  and 
phone  calls  during  a  break. 


One  speaker  at  a  time,  please.  Sidebar 
conversations  make  it  difficult  for  others 
to  hear. 
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Some  Logistics  and  Guideiines 


have  a  hard  time 
asking  for  help. 


Get  Your 
ASK 
In  Gear! 
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Agenda 


Start 

End 

Duration 

9:00 

9:30 

0:30 

9:30 

10:30 

1:00 

10:30 

10:45 

0:15 

10:45 

11:30 

0:45 

11:30 

12:30 

1:00 

12:30 

2:00 

1:30 

2:00 

3:00 

1:00 

3:00 

4:00 

1:00 
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Activity 


Strategic  Planning  Overview 

Presentation 

Break 

Presentation  &  Group  Exercise 
Lunch 

Group  work  session 
Outbrief 
More  practice 

Times  are 
approximate 
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Module  Topics 


Introductory  concepts 

What  is  a  “performance  measure”? 


What  do  we  measure? 


The  relationship  of  performance  measurement  to  work  plans 


What  could  possibly  go  wrong? 

What  does  a  “good”  performance 
measure  look  like? 

Putting  it  all  together 
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Why  We  Are  Here 


Accountability  for  performance  has  become  increasingly  critical. 

■  The  VA  must  understand  the  key  drivers  behind  its  performance  and 
demonstrate  the  value  of  its  work. 

■  Focus  on  the  use  of  data  is  a  key  principle  guiding  any  performance 
management  system. 

■  Performance  measurement  provides  information  on  how  current  programs 
and  activities  are  working  and  how  resources  can  be  allocated  to  optimize 
effectiveness. 

Performance  management  strives  to  align 
all  the  VA’s  subsystems  to  achieve  results. 

■  Divisions 

■  Processes 

■  Teams 

■  Individuals 
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You  Can’t  Manage  What  You  Can’t  Measure 


Performance  management  A  forward-looking  process  used  to  set  goals 

and  regularly  check  progress  toward 
achieving  those  goals. 

Performance  measurement  A  process  by  which  an  organization  monitors 

important  aspects  of  its  programs,  systems, 
and  processes. 


Performance  measure 


The  quantitative  or  qualitative  measure  that 
is  designed  to  track  progress  towards 
achieving  a  goal. 


Lcraiej 
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Level  of  Performance 


We  All  Want  To  Improve  Our  Performance 


>• 

Time 


Q. 

CD 


O 


<D 

O 

C 

CD 


‘t 


0 

CL 


•  Cost 

•  Quality 

•  Time 


What  can  we  do  as  individuals  and  as  teams  of 
individuals  to  realize  our  potential? 
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But  First  You  Need  to  Determine  Where  You  Are 


In  the  absence  of  measurement, 
we  are  like  a  traveler  without  a 
map  ...  wandering, 
directionless,  and  confused. 

But  to  use  a  map,  you  must 
know  where  you  are! 
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Measurement  helps  us  establish  where  we  are,  whether  we’ve 
made  progress,  what  has  changed,  and  where  we  need  to  go. 
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The  World  Without  Measurement 


Science? 


Engineering? 


Medicine? 
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Without  Performance  Measurement 


People 
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Building  Aiignment  and  Synergy 


Performance  management  strives  to  align 
all  the  VA’s  subsystems  to  achieve  results. 

■  Divisions 

■  Offices 

■  Teams 

■  Individuals 


Time 

- ► 
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Cascading  and  Linked  Goais 


Increasingly 

strategic 


L 


/I 


VA  strategic  Plan 


] 


VHA  Strategic  Plan 


OIA  Strategic  Plan 


OIA  Operating  Plan 


Division  Operating  Plan 


D 

1 


Office  Work  Plans 


] 


Individual  Performance  Plans  ] 


Increasingly 

detailed 
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OIA  Performance  Management 


OIA  Goals 


Division  Initiatives 


Some  divisions  develop  initiatives 
that  are  mapped  to  OIA  goals. 

Accomplishing  a  division  initiative 
supports  the  achievement  of  the  OIA 
goals. 
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19 


OIA  Performance  Management 


OIA  Goals 

Division  Initiatives 

Work  Plan 

In  some  cases,  work  plan  activities 
are  mapped  directly  to  OIA  Goals. 


A  Division  Office  establishes  a 
Work  Plan  to  address  the  intent 
of  a  Division  Initiative. 

If  the  plan  is  executed 

successfully,  then 

the  intent  of  the  initiative  is  met. 

If  the  intent  of  the  initiative  is 
met,  then  progress  is  made  in 
achieving  the  OIA  goal  that  the 
initiative  is  mapped  to. 
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OIA  Performance  Management 


OIA  Goals 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

Division  Initiatives 

Work  Plan 


It  is  at  the  work  plan  level 
where  work  is  actually 
accomplished. 

A  work  plan  is  composed  of 
activities. 

Activities  have  outcome(s). 

If  the  outcomes  are 
successfully  achieved  then  the 
work  plan  was  successfully 
achieved  ...  which  ultimately 
leads  to  achievement  of  the 
OIA  goals. 
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OIA  Performance  Management 


OIA  Goals 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

Division  Initiatives 

Work  Plan 


Activity  #1  Activity  #2  Activity  #n 


Outcomes  are  the  actualization  of 
the  activities  documented  in  the 
work  plan. 

Outcomes  are  observable, 
measurable  entities. 

Outcomes  are  what  we  care 
about. 


Outcomes  are  what 
we  measure! 
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Module  Topics 


Introductory  concepts 
►  What  is  a  “performance  measure”? 


What  do  we  measure? 


The  relationship  of  performance  measurement  to  work  plans 


What  could  possibly  go  wrong? 

What  does  a  “good”  performance 
measure  look  like? 

Putting  it  all  together 
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Clarifying  Some  Terminology 


Term 

Explanation 

Activity 

A  description  of  the  task  that  produces  an  intended 
outcome.  The  outcome  is  the  objective  of  the  activity  and 
should  be  specified.  The  outcome  is  not  the  output.  It  is  the 
hoped-for  condition  that  arises  based  on  the  output. 

Performance 

Measure 

Quantitative:  A  quantitative  variable  that  portrays  a  key 

characteristic  of  the  outcome  and  the  scale  it 
applies  to. 

Qualitative:  A  categorical  variable  that  portrays  a  key 
characteristic  of  the  outcome. 

Target 

This  is  the  target  value  for  the  performance  measure.  The 
target  is  the  value  of  the  performance  measure  that  is  to  be 
achieved  AND  the  date  when  it  will  be  achieved. 

© 


Always 


strive  for  clarity  when  writing  each  of  these. 
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Two  Types  of  Measures 


Quantitative 


Qualitative 
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Quantitative  Performance  Measures 


A  quantitative  performance  measure  is  composed  of 

•  a  number 

•  a  unit  of  scale 

The  number  provides  the  magnitude  (how  much) 

The  unit  of  scale  gives  the  number  a  meaning  (what). 
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Examples:  Numbers  and  Unit  of  Scale 


Number 

3 

27 

150 

10.5 

9 

220 


Unit  of  Scale 

Dollars 

Certifications 

Patients 

Centimeters 

Defects 

Minutes 
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Quantitative  Performance  Measures 


In  many  cases,  multi-dimensional  units  of  measure  are  used. 

•  Expressed  as  ratios 

•  Examples: 

#  of  accidents  per  year 

#  of  certifications  awarded  per  #  of  individuals  trained 

#  of  successful  logins  to  system  per  login  attempts 


X 


Qualitative  Measures 


Good  Average  Poor 


Used  when  quantitative  measures 
are  unavailable  or  inapplicable. 
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Qualitative  Performance  Measures 


Qualitative  measures  are  also  referred  to  as  categorical  data. 

Qualitative  indicators  are  viewed  as  subjective  that  are  filtered  through 
perceptions.  They  can  therefore  be  imprecise  when  compared  to 
quantitative  measures. 


Document  development  New  Technology  Introduction  Customer  Satisfaction 


Plan  completed 
Research  completed 
Draft  complete 
Draft  peer-reviewed 
Completed/Approved 


Requirements  defined 
Design  completed 
Prototype  developed 
Testing  completed 
Rollout  completed 


Excellent 
Very  good 
Good 

Satisfactory 

Poor 

Very  poor 
Unacceptable 
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Module  Topics 


Introductory  concepts 

What  is  a  “performance  measure”? 

►  What  do  we  measure? 

The  relationship  of  performance  measurement  to  work  plans 


What  could  possibly  go  wrong? 

What  does  a  “good”  performance 
measure  look  like? 

Putting  it  all  together 
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What  Do  We  Measure? 


Work  Plan 


Activity  #1  Activity  #2  Activity  #n 


Quality  of  outcome 


Progress  towards 
achieving  outcome 
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When  you  think 
performance  measurement, 
think  ... 


on  the  customer. 


positive 

impact 


Work  plan  activity 


Has  at  least  one 


Intended  to  benefit 


Outcome 


Internal  or  external  customer 


The  best  performance  measures 
evaluate  the  impact  of 
the  outcome  of  the  activity 
on  the  customer. 


Define  your  performance 
measure  from  the 
customer’s  perspective! 


And  what  matters  to  the  customer? 
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What  Do  We  Mean  By  “Quality”? 


In  technical  usage,  quality  can  have  two  meanings: 

the  characteristics  of  a  product  or 
service  that  bear  on  its  ability  to  satisfy 
stated  or  implied  needs 


Crosby 


Juran 


a  product  or  service  free  of  deficiencies 


Source:  The  American  Society  for  Quality;  Quality  Glossary;  http://asq.0rg/gl0ssary/q.html 

ASO 
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Comparing  the  Definitions  of  Quaiity 


Cj  Features  that  meet  or 
"  exceed  customer  needs 


Freedom  from  deficiencies 


Higher  quality  enables  an  organization  to: 

■  increase  customer  satisfaction 

■  meet  or  exceed  competition 

■  increase  market  share 

■  secure  premium  prices 


Higher  quality  enables  an  organization  to: 

■  reduce  rework  and  waste 

■  reduce  field  failures 

■  reduce  customer  dissatisfaction 

■  reduce  pilot  testing  time 

■  shorten  time  to  put  new  products  on  market 

■  increase  capacity 

■  improve  delivery  performance 
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A  staff  person  in  your  division  has  been  tasked 
to  develop  a  training  course  to  teach  veterans 
how  to  log  onto  a  VA  service. 

Which  of  the  following  performance  measures 
is  the  best . . .  and  why? 

1 .  The  number  of  pages  of  materials  prepared  for  the  training  course. 

2.  The  number  of  students  that  have  registered  for  the  training. 

3.  Course  evaluations  -  the  percentage  of  training  course  attendees  that 
rated  the  course  as  “satisfactory”  or  “excellent.” 

o  4.  The  percentage  of  veterans  that  were  able  to  log  on  to  the  HI  service 
without  help  from  the  call  center  (after  receiving  the  training). 

5.  Post-course  survey  results  of  veterans  that  took  the  training. 


Software  Engineering  Institute  Carnegie  Mellon  University 


Developing  Effective  Performance  Measures 
Mark  Kasunic,  October  14,  2014  37 

©2014  Carnegie  Mellon  University 


Example:  Bad  Performance  Measure 


Perf.  Meas.  =  #  of  pages  materials  prepared  for  the  training  course. 


Measuring  performance  from 
the  organization’s  point  of 
view,  rather  than  from  the 
customer’s  perspective. 


OPZNTON  OF  OMR 

P0OOUCT 
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Guidelines:  Outcome-Based  Performance  Measures 


■  Be  careful  to  distinguish  an  activity  from  the  outcome  of  the 
activity.  Performing  a  task  or  a  series  of  tasks  is  not  an  outcome! 

■  Ask  yourself:  How  did  the  product  or  service  impact  the  customer. 

■  So,  don’t  define  a  performance  measure  from  your  perspective. 
Write  it  from  the  customer’s  perspective! 


What  Do  We  Measure? 


Work  Plan 


Activity  #1  Activity  #2  Activity  #n 


Software  Engineering  Institute 


Carnei 


Quality  of  outcome 


^  Progress  towards 
achieving  outcome 
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When  Developing  a  Product  -  Measure  Progress 


Setting  a  target  date  for  your  commitment  allows  you  to  look  forward  and 
effectively  plan  next  steps. 


Measuring  schedule  performance  using  interim  milestones  provides  an  early 
warning  indicator  that  alerts  you  if  the  project  is  slipping. 


Understanding  schedule  progress 
allows  you  to  take  corrective  actions 
(a  recovery  plan)  so  that  you  meet 
your  commitment. 

Schedule  performance  can  be  measured 
using: 

■  a  quantitative  measure 

or 

■  a  qualitative  measure 
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Quantitative  Example:  Earned  Value 


Estimate  at 

Completion  (EAC) 

Cost  Variance 

Actual  Cost  of  Work 

Variance  at 

(CV)  at  time 

Performed  (ACWP) 

Completion  (VAC) 

(a.k.a.  EV) 

(a.k.a.  EarnedValue  (EV)) 

^  I 

Cost 


Budget  at 
Completion  (BAC) 

Budgeted  Cost  of  Work 
Scheduled  (BCWS) 

(a.k.a.  Planned  Value  (PV)) 

Schedule  Variance 
(SV)  at  time 

Budgeted  Cost  of  Work 
Performed  (BCWP) 


Time 
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Qualitative  Example:  Milestones  Completed 


PROJECT  CALENDAR 


PROJECT  START 


MILESTONE  1 


MILESTONE  2 


MILESTONE  3 


PROJECT  END 
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Example:  Schedule-Based  Qualitative 

Performance  Measure 


New  Technology  Introduction  Target  Milestone  Date 


Requirements  defined 
Design  completed 
Prototype  developed 
Testing  completed 
Rollout  completed 


Oct.  22,  2014 
Nov.  30,  2014 
March  1,2015 
April  1,2015 
August  15,  2015  ~j-- 


Used  to  track  progress 
in  project  plan. 

Target  criterion 
expressed  in  project  plan 
and  defined  as  part  of 
performance  measure. 


Software  Engineering  Institute  Carnegie  Mellon  University 


Developing  Effective  Performance  Measures 
Mark  Kasunic,  October  14,  2014  44 

©2014  Carnegie  Mellon  University 


Module  Topics 


Introductory  concepts 

What  is  a  “performance  measure”? 


What  do  we  measure? 


^  The  relationship  of  performance  measurement  to  work  plans 


What  could  possibly  go  wrong? 

What  does  a  “good”  performance 
measure  look  like? 

Putting  it  all  together 
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Controlling  Your  Destiny 

■  To  control  the  way  you  work,  you  must  plan  your  projects. 

■  For  management  to  trust  these  plans,  you  must  make  accurate  plans. 

■  To  make  accurate  plans,  you  must  have  data. 

■  To  have  data,  you  must  measure  your  performance. 


INTtGkftTION 


transformation 


ANALYSIS 


\ 

exploratioi 


Management  101 


Performance  measures  must  be  part  of  every 
work  plan. 

The  measures  should  be  tracked  closely  by 
the  individual  who  is  managing  the  work  effort. 
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Tracking  Status 


If  you  cannot  recognize  a  one-day  slip,  you  can  only  take  action  when 
the  delay  is  obvious. 


But,  by  then,  it  is  usually  too  late 
to  recover. 


.  .schedule  slips,  one  day  at  a  time.  ” 

-  Fred  Brooks  in  The  Mythical  Man  Month 
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Relationship  of  Pians  to  Performance  Measurement 


Office  work  plan  tracking  of 
performance  measures 
(e.g.,  daily  or  weekly  tracking) 


Quarterly  snapshots 
of  performance  measures 
are  submitted  to  the 
performance  measurement 
tracking  system 
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Module  Topics 


Introductory  concepts 

What  is  a  “performance  measure”? 


What  do  we  measure? 


The  relationship  of  performance  measurement  to  work  plans 
►  What  could  possibly  go  wrong? 


What  does  a  “good”  performance 
measure  look  like? 

Putting  it  all  together 
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When  Performance  Measurement  Goes  Bad 


Laziness 

Vanity 

Narcissism 

Too  Many 

Pettiness 

Inanity 
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Laziness 


Measuring  what  is  easy  to 
measure  ...  instead  of  what  is 
important  to  measure. 


a.k.a. 

The  Measurement  Inversion 
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Measurement  Inversion 


Selection 

Priority 


High-value  measures 


Low-value  measures 


7J 

<D 

0 

< 

fi) 

3 

O 

<D 


Low 
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Measurement  Inversion 


Typical  ■  Time  spent  in  an  activity 

■  #  documents  developed 

,  ,  ■  Attendance  at  Quality  Improvement  training 

Low-value  measures 

■  Near-term  costs  of  a  project 

■  #  of  violations  found  in  safety  inspections 


Often  Ignored 


measures 


■  Value  of  an  activity 

■  Impact  based  on  purpose  of  document 

■  Effect  of  training  on  #  product  defects 

■  Long-term  benefits  of  a  project 

■  Reduction  in  risk  of  catastrophic  accidents 
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Vanity 


Gaming  the  System  ... 

The  sin  of  using  metrics 
that  you  have  mastered, 
or  that  make  you  iook 
good,  rather  than  heip 
drive  improved 
performance  resuits. 
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Narcissism 


Measuring  performance  from  the 
organization’s  point  of  view,  rather 
than  from  the  customer’s 


perspective. 


_ ^  Developing  Effective  Performance  Measures 
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56 


«  r 


Too  Many  Measures 


Too  many  measures  tends  to  distract 
management  from  those  most  critical 
to  managing  the 
organization. 
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You  Need  Data  To  Manage 


GOOD 

Metrics  are  good. 


But ... 
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This  Is  A  Data  Collection  Faliacy 


GOOD 

Metrics  are  good. 
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^  Inanity 


Some  organizations  create  metrics 
without  any  consideration  for  the 
consequences. 


oua  GOAL  I5>  10  LJlVrTL 
BO&-f=aEE:  SOFTWARE, 
ILL  m  ATEW-OOLLAR 
BOMU5  FOR  EVEHV  QUG 

you  find  and  Fiy.. 


I  HOPE 
THIS 
OKLVES 
TME  RIGHT 

behavior 


VtA  gohna 

L0RITE.ME  A 
NEtO  ^MIVAM 
THIS  AFTER¬ 
NOON^ 
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Pettiness 


Organizations  sometimes  focus  on  a 
smali  part  of  what  matters,  rather  than 
on  the  totality,  or  end  results. 
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Module  Topics 


Introduction  to  performance  management 

When  performance  measurement  goes  bad 
►  What  does  a  “good”  performance  measure  look  like? 

Putting  it  all  together:  Developing  effective  performance  measures 
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What  Is  a  “Good”  Performance  Measure? 


6QAL  SETTfMG 


sfsrarrc 

urASURAbie 

"  ATtaiwabz-e: 

RacvAM’r 

TrMb'6oa^ij 


Well-formed  performance  measures  are  the 
answer  to  the  “M”  part  of  SMART. 

A  performance  measure  provides 
objective  evidence  of  the  degree 
to  which  a  performance  result  is 
occurring  over  time. 
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The  intended  measure  does  not  seem  to  be  measuring 
the  outcome  of  the  described  activity. 

Instead,  it  is  measuring  the  implementation  of  an 
already  developed  program. 


Activity 


Measure 


Develop-  risk-management-  program- for-DoD/VA-  Produce-Risk- Management-  Reports-  Monthly  -  and-Resolve- 

Information- Sharings  risks-within-target- dates-  80%-of-the-timea 


The  outcome  of  this  activity  is: 

•  A  risk  management  program. 

What  is  the  “program.”  Is  it  a  process?  A 
database?  A  service?  All  of  the  above? 


Is  this  a  performance  measure?  It  is 
expressed  as  an  activity.  “Produce  ...” 

This  is  also  a  compound  statement: 

(a)  produce  reports 

(b)  resolve  risks 
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Some  Pretty  Good  Examples  - 1 


Department  of  Agriculture 

Program:  Rural  Utilities  Service  Telecommunications  Program 


Measure 

Percentage  of  rural  telecommunications  subscribers 
receiving  new  or  improved  service 


Explanation 

The  purpose  of  this  program  is  to  improve  and  expand  telecommunications  in  rural  areas. 
The  measure  directly  relates  to  this  purpose.  Once  targets  are  established,  this  measure 
will  indicate  whether  the  program  is  effective.  In  addition,  over  time  this  measure  will 
provide  information  on  the  portion  of  rural  America  benefiting  from  program  loans. 

The  measure  supports  the  agency’s  mission  to  support  rural  America.  Performance  is 
measured  using  a  formula  based  on  the  number  of  subscribers  supported  by  new  loan 
funds  divided  by  the  total  number  of  rural  subscribers  of  active  borrowers  (both  are  based 
on  data  reported  by  borrowers).  The  data  is  verifiable  since  borrower  records  on  the 
number  of  customers  could  be  used  to  verify  the  numbers  reported  to  USDA. 
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Some  Pretty  Good  Examples  -  2 


Department  of  Commerce 
Program:  Patent  and  Trademark  Office 


Measure 

Cost  per  patent  disposed 


The  outcome  they 
are  interested  in 
is  “improved 
efficiency.” 


Explanation 

A  primary  measure  of  the  efficiency  of  processing  patents  is  the  cost  per  patent  disposed. 

The  measure  is  calculated  by  dividing  total  PTO  expenses  associated  with  the 
examination  and  processing  of  patents  (including  associated  overhead  and  support 
expenses)  by  production  units. 


Some  Pretty  Good  Examples  -  3 


Department  of  Education 
Program:  Troops-To-Teachers 


Activity  ...  but  it’s 
referring  to  the 


outcome  that  they 
are  interested  in. 


Measure 

Percentage  of  Troops-To-Teachers  who  remain  in  teaching  for  three  or  more  years 
after  placement  in  a  teaching  position  in  a  high-need  school. 

Explanation 

The  Troops-to-Teachers  program  recruits,  prepares  and  places  retired  military  personnel 
as  teachers  in  elementary  and  secondary  schools  in  "high-need"  school  districts. 

One  of  the  Department’s  highest  priorities  is  to  improve  teacher  quality  across  the 
country,  especially  in  school  districts  serving  a  large  proportion  of  students  from  low- 
income  families.  . . 


This  is  the  outcome  of 
interest.  This  is  what’s 
measured! 


This  measure  is  a  useful  indicator  of  the  program's  effectiveness  in  its 
two  key  outcomes:  placement  and  persistence. 


Program  information  is  collected  from  participating  State  and  regional  offices  which  are 
required  by  statute  to  report  on  retention  rates  of  their  program  completers. 


Some  Pretty  Good  Examples  -  4 


Department  of  Health  and  Human  Services 
Program:  Foster  Care 


Measure 

The  percentage  of  children  with  substantiated  reports  of  maltreatment  that  have  a 
repeated  report  within  six  months. 


Explanation 

The  purpose  of  this  program  is  to  help  States  provide  care  for  children  who  are  under  the 
jurisdiction  of  the  administering  State  agency  and  need  temporary  placement  outside  their 
homes,  in  a  foster  family  home  or  an  institution.  HHS  provides  funds  to  States  to  assist  with:  the 
costs  of  foster  care  maintenance  for  eligible  children;  administrative  costs  to  manage  the 
program;  and  training  for  staff,  for  foster  parents  and  for  certain  private  agency  staff. 

This  measure  shows  the  system's  relative  success  at  preventing  child  abuse  by  highlighting 
when  it  fails  to  do  so. 

The  data  comes  from  HHS’  data  system-- Adoption  and  Foster  Care  Analysis  and  Reporting 
System,  and  is  verifiable  and  reliable. 

Desired  outcome. 
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When  Performance  Measurement  Goes  Good 


Relevant 


Avoids  perverse  incentives 
Well-defined 


)  Timely 


Reliable 


3  Comparable 


Verifiable 
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Testing  a  Measure  Using  the  Goodness  Criteria 


Criteria 

Questions 

Relevant 

■  Does  the  measure  attempt  to  capture  success  around  a  key  OIA 
objective? 

■  What  does  it  tell  you  about  how  your  are  performing? 

■  Does  it  accurately  represent  what  you  are  trying  to  assess? 

Avoids  perverse 
incentives 

■  Does  the  measure  encourage  unwanted  behavior  (e.g.,  not 
reporting  mistakes)? 

■  Could  you  improve  the  measure  without  really  improving  actual 
performance? 

■  Does  it  allow  innovation?  Or  does  it  discourage  improvements 
to  service  delivery? 

Well-defined 

■  Is  the  measure  expressed  clearly  so  that  it  is  easily  understood? 

■  Does  it  have  an  unambiguous  definition,  so  it  can  be  collected 
consistently? 

■  Can  you  gather  all  the  data  or  evidence  you  need  to  produce 
the  measure? 
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Testing  a  Measure  Using  the  Goodness  Criteria 


Criteria 

Questions 

Timely 

■  Does  the  measure  provide  information  in  time  for  action  to  be 
taken? 

■  What’s  the  lag  between  the  event  and  information  becoming 
available? 

Reliable 

■  Is  the  performance  measure  accurate  enough  for  its  use? 

■  Has  the  measure  been  validated  as  a  useful  indicator  by  your 
manager? 

■  Will  the  measure  change  because  of  issues  outside  your 
control? 

Comparable 

■  Does  the  measure  allow  comparison  with  past  performance? 

■  Does  it  allow  comparison  with  other  agencies  delivering  a 
similar  service? 

Verifiable 

■  Does  documentation  exist  so  that  the  process  behind  the 
measure  can  be  validated? 

■  Given  the  documentation,  could  an  objective  outsider  come  up 
with  the  same  results? 
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Module  Topics 


Introductory  concepts 

What  is  a  “performance  measure”? 


What  do  we  measure? 


The  relationship  of  performance  measurement  to  work  plans 


What  could  possibly  go  wrong? 

What  does  a  “good”  performance 
measure  look  like? 

►  Putting  it  all  together 
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Planning  Your  Performance  Measure 


Performance  Measurement  Planning  Worksheet 

Part  I.  New  Product  Development 


Qualitv-Based  Performance  Measure 

Activitv 

Describe  what  you  will  do. 

Actrvitv  Outcome 

What  will  be  the  tangible  result  of  what  you  do?  What  is  the  product? 

Customer 

Who  will  use  or  benefit  from  your  product? 

Quality  end  Impact 

How  will  the  customer  benefit  from  your  product? 

What  concerns  might  the  customer  have  about  this  product? 

What  are  the  most  important  characteristics  of  the  product  that  the 
customer  will  care  about? 

How  will  the  customer  judge  the  quality  of  this  product? 

How  wiff  you  assess  the  quality  of  what  you  are  delivering  to  the 
customer? 

Is  your  quality  performance  measure  quantitative  or  qualitative? 

If  you  are  using  a  qualitative  measure,  what  are  the  categorical 
variables  that  will  characterize  the  quality  of  your  product? 

Perf.  Meas. -Quality 

Write  the  definition  for  your  quality-based  performance  measure.  Be 
su  re  to  i  ncl  ude  the  unit  of  sea  I  e  for  the  measu  re. 

Oats  Collection 

Where  will  you  obtain  the  performance  measurement  data? 

How  will  you  collect  the  data  you  need  for  your  perform  am  ce  measure? 

How  often  will  you  collect  the  performance  measurement  data? 

Relevance  Check 

How  does  your  performance  measure  capture  the  most  important 
aspects  of  the  product's  objective? 

Target  for  Quality 

Describe  the  value  of  the  quantitative  or  qualitative  performance 
measure  at  the  end  of  FV2015  or  the  date  if  completed  before  the  end 
of  FY2015. 

Soii^wArt  En9irt««fiilg  InnHute  t  nlv^-iviiv  '  _  '  |  ltif3 
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Let’s  Practice  Together 


Software  Engineering  Institute 
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The 


The  Need: 

35%  of  young  veterans  (18-24  year  olds)  are  unemployed.  Economists  cite  a  lack  of  marketable 
civilian  skills  and  the  need  for  education  degrees,  vocational  certifications  and  experience  in 
professional  settings. 

Intervention: 

Provide  support  to  veterans  in  completing  training  programs  by: 

■  Locating  appropriate  programs 

■  Securing  financial  aid 

■  Providing  tutoring  resources  and  internship  placements 

Additional  Information: 


The  “young  veterans”  were  interviewed  about  the  proposed  intervention.  Two  concerns  were 
expressed.  Young  veterans  feel  that  they  need  to  make  a  minimum  of  $1 5/hour  to  support  their 
needs.  Many  responded  that  they  currently  have  limited  access  to  a  computer. 

Education  records  of  the  veterans  are  updated  regularly  in  the  Agency’s  database.  Financial  aid 
records  are  accessible  through  the  Department  of  Education.  Employment  status  is  tracked  by  the 
Agency  on  a  weekly  basis  and  recorded  in  the  Agency’s  database. 
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Part  2.  Service  Delivery 


Quality- Based  Performance  Measure 

Activity 

Describe  what  you  will  do. 

Activity  Outcome 

What  will  result  from  the  service  that  you  provide? 

Customer 

Who  will  use  or  benefit  from  your  service? 
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Part  2.  Service  Delivery 

m 


Quality- Based  Performance  Measure 

Quality  and 

Impact 

How  will  the  customer  benefit  from  your  service? 

What  concerns  might  the  customer  have  about 
your  service? 

What  are  the  most  important  characteristics  of 
the  service  that  the  customer  will  care  about? 

How  will  the  customer  judge  the  quality  of  your 
service? 

How  will  you  assess  the  quality  of  the  service 
that  you  deliver  to  the  customer? 

Is  your  quality  performance  measure 
quantitative  or  qualitative? 

' 

If  you  are  using  a  qualitative  measure,  what  are 
the  categorical  variables  that  will  characterize 
the  quality  of  your  prod  uct? 
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Quality-Based  Performance  Measure 

Perf.  Meas.  - 
Quality 

Write  the  definition  for  your  quality-based 
performance  measure.  Be  sure  to  include  the 
unit  of  scale  for  the  measure. 

' 

Data  Collection 

Where  will  you  obtain  the  performance 
measurement  data? 

How  will  you  collect  the  data  you  need  for  your 
performance  measure? 

How  often  will  you  collect  the  performance 
measurement  data? 

Relevance  Check 

How  does  your  performance  measure  capture 
the  most  important  aspects  of  the  product's 
objective? 

Target  for  Quality 

Describe  the  value  of  the  quantitative  or 
qualitative  performance  measure  at  the  end  of 
FY2015  or  the  date  if  completed  before  the  end 
of  FY2015. 
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Transferring  The  Data 


Performance  Measurement  Planning  Worksheet 

Part  I.  New  Product  Development 

OtiaHty-tatca  rertotmance  Meatute 

OeKKtw  wtiat  you  do 

what  wM  tt>«  tan^M  result  of  w^at  you  dot  WtvM  t»  tt>e  proUuaT 
■  iAto.T“  Who  wM  use  Of  ban*af  from  your  Dnidurtt 

Howr  wfl  the  (ustomar  banafil  Irom  yoiv  pfoduct^ 

What  concerns  nyght  the  customer  have  about  this  product^ 


How  wM  the  customer  judie  the  auaiity  ol  this  product’ 
NOW  wW  you  otttu  the  owotty  bf  whof  yow  ore  deAvennp  to 


Is  vour  euatey  pe»form«r>ce  measure  quanwative  or  qualitatNet 
If  you  arc  using  a  Qualitative  measure,  what  are  the  cetegorical 
vanabses  that  will  charactenie  the  Quality  of  your  product? 


Where  will  you  obtam  the  performance  meesurement  date? 

Mow  w«  you  co*c<t  the  dau  you  need  for  vour  performance  measure? 
How  oftrn  wQI  you  coOect  Ihr  prrlormancr  mcawirrmeol  data’ 

How  does  your  performsote  measure  capture  the  most  enportant 
aspects  of  the  product’s  obfcc the’ 

Describe  the  vahie  Of  the  Quantitative  or  Qoaiitatnre  performance 
measure  at  the  end  of  fvjOlS  or  the  date  if  completed  before  the  end 
efrrMis 


▼ 

▼ 

▼ 

▼ 

▼ 

▼ 

Cffice  ' 

Activity  ^ 

Start  ' 

(mm/ww) 

End 

jmm/yyyy} 

Measure  ^ 

Target  ^ 

Frequency  ^ 

Data  Source  ^ 

Measure  Owner’' 

Data  Entry  ^ 

Comments 

Performance  Measurement  Spreadsheet 
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Current  Performance  Measurement  Template 


Term 

Definition 

Office 

Division’s  office. 

Activity 

A  succinct  description  of  the  task  being  measured,  including  a  clear  outcome. 

Start  (mm/yyyy) 

Date  you  will  start  measuring. 

End  (mm/yyyy) 

Date  the  target  should  be  achieved. 

Measure 

A  scaled  quantitative  or  qualitative  variable  that  represents  an  important  attribute 
of  the  activity  being  measured. 

Target 

A  particular  value  for  the  performance  measure  that  is  to  be  accomplished  by  a 
specific  date  in  the  future.  Both  the  value  and  the  date  must  be  specified. 

Frequency 

The  frequency  that  you  track  the  measure. 

Data  Source 

The  source  of  the  measurement  data. 

Measure  Owner 

Name  of  individual  responsible  for  all  activity  related  to  this  measure.  Provides 
the  data  needed  to  establish  and  maintain  the  measure. 

Data  Entry 

Name  of  person  doing  the  data  entry  in  the  online  tool. 

Comments 

Additional  comments  of  clarification  if  needed. 
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Mapping  Worksheet  Results  to  PM  Template 


Planning  Worksheet  Section 

Performance  Management  Template 

Office 

Activity  and  Activity  Outcome 

Activity 

Start  (mm/yyyy) 

Completion  Date 

End  (mm/yyyy) 

■  Perf.  Meas.  -  Quality 

■  Perf.  Meas.  -  Schedule 

Measure 

■  Target  for  Quality 

■  Target  for  Schedule 

Target 

Data  Collection  (How  often  will  you  collect ...) 

Frequency 

■  Data  Collection  (Where  will  you  obtain  ...) 

■  Tracking  Tool 

Data  Source 

Measure  Owner 

Data  Entry 

Comments 

_ ^  Developing  Effective  Performance  Measures 
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Reminder:  Relationship  of  Work  Plan  to 

Per.  Meas.  System 


Division  work  plan  tracking  of 
performance  measures 
(e.g.,  daily  or  weekly  tracking) 


Quarterly  snapshots 
of  performance  measures 
are  submitted  to  the 
performance  measurement 
tracking  system 
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Weekly  Tracking  of  Performance  Measure 


Planned  progress 


10-14-2014 

t 

t 

t 

09-01-2015 

1®‘  Quart. 

2^^^  Quart. 

3'"'^  Quart. 

Date 
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1 .  Break  out  into  your  groups. 

2.  Each  group  designates 

•  Facilitator 

•  Time  keeper 

•  Spokesperson 

3.  Use  the  performance  measurement  planning 
template  to  organize  your  work. 

4.  Group  decides  when  to  take  a  break. 

5.  If  you  complete  development  of  a  performance  measure,  start  in  on  an 
additional  one.  Keep  on  rolling,  but  be  prepared  to  report  out  on  one  of 
your  performance  measures  during  the  outbrief. 
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Contact  Information 


Mark  Kasunic 

Senior  Member  of  Technical  Staff 
Software  Engineering  &  Acquisition  Practices 
Customer  Technical  Solutions 
Telephone:  +1  412-268-5863 
Email:  info@sei.cmu.edu 


U.S.  Mail 

Software  Engineering  Institute 
Customer  Relations 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-2612 
USA 


Web 

www.sei.cmu.edu 

WWW.  se  i .  cm  u .  ed  u/co  ntact .  cf  m 


Customer  Relations 

Email:  info@sei.cmu.edu 
SEI  Phone:  +1  412-268-5800 

SEI  Fax:  +1  412-268-6257 
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